A differential correction based shadow removal method for real-time monitoring

Shadow removal is an important issue in the field of motion object surveillance and automatic control. Although many works are concentrated on this issue, the diverse and similar motion patterns between shadows and objects still severely affect the removal performance. Constrained by the computational efficiency in real-time monitoring, the pixel feature based methods are still the main shadow removal methods in practice. Following this idea, this paper proposes a novel and simple shadow removal method based on a differential correction calculation between the pixel values of Red, Green and Blue channels. Specifically, considering the fact that shadows are formed because of the occlusion of light by objects, all the reflected light will be attenuated. Hence there will be a similar weakening trends in all Red, Green and Blue channels of the shadow areas, but not in the object areas. These trends can be caught by differential correction calculation and distinguish the shadow areas from object areas. Based on this feature, our shadow removal method is designed. Experiment results verify that, compared with other state-of-the-art shadow removal methods, our method improves the average of object and shadow detection accuracies by at least 10% in most of the cases.


Introduction
With the application of intelligent video surveillance and automatic control, shadow removal becomes more and more important. An effective shadow removal method can minimize the interference of shadows on object detection, recognition and control [1,2]. In fact, shadow, as a phenomenon due to light being blocked by object, has the same motion property with object itself. Therefore, it is difficult to identify and remove shadows based on the judgment of motion property of image areas in the video. Meanwhile, considering the computational efficiency in real-time application and the cost of real-time monitoring equipment, deep neural network based methods [3][4][5][6][7] are difficult to be widely used. Hence, shadow removal is still an interesting and challenge work for real-time monitoring. To satisfy the need for real-time computing in object surveillance, background difference based methods [8,9] are still the most cost-effective methods in practical application. After background difference, all the areas of objects with shadows are detected. Then, the shadows can be further removed based on one or more features which can well distinguish shadows from objects. Currently, the best feature is the RGB pixel ratio [10]. As illustrated in [10], RGB pixel ratios of shadow areas before and after shadow covering are similar, and are significantly different from the ones of object areas. Hence, the shadow areas can be found by RGB pixel ratio comparison (RGB-PRC) method. However, according to the principle of shadow formation, it is the occlusion of light by objects that forms the shadow. In fact, it cannot be sure that the same areas in the images with and without shadows follow the same ratios of RGB pixel values, especially when the light is severely occluded. This phenomenon will be discussed in Section 3. To conquer this problem, this paper investigates all the image data in ISTD dataset [3,4], and discovers a new pixel feature in the shadow areas of images according to a differential correction calculation. It is named RGB pixel differential correction (RGBP-DC) feature. Further, a new differential correction based shadow removal (DC-SR) is proposed according to the aforementioned RGBP-DC feature. The experiment results show that our DC-SR outperforms the state-of-the-art shadow removal methods.
The organizations of the remainder of this paper are structured as follows. In Related Work Section, the related works are described. In RGB Pixel Differential Correction Feature Section, the limitation of RGB-PRC method is illustrated, and a new RGBP-DC feature is proposed. Then, a new DC-SR method is designed based on the RGBP-DC feature in Differential Correction Based Shadow Removal Method Section. In Experiments Section, a lot of comparison experiments are performed to verify the effectiveness of the proposed DC-SR method. Finally, the conclusion and limitation of the proposed method are given in Conclusion Section. The main contributions of our work are summarized as follows: • A new pixel feature, i.e., RGBP-DC feature, is found in the shadow areas of images.
• A new differential correction based shadow removal (DC-SR) method is proposed.

Related work
Currently, in the field of real-time monitoring, we still need the background difference to quickly find out the areas of objects. After that, the goal of shadow removal becomes to distinguish the shadow areas from the object areas based on two kinds of methods: the model-based and feature-based methods, respectively.
Model-based methods mainly use prior information to train corresponding models. For example, Zhang proposed a robust vehicle detection method with shadow elimination [11]. Amin Benish proposed a shadow mask extractor by using a three color attenuation model (TAM) and intensity information to segment the shadow area [12]. Saritha Murali proposed a method to remove shadows from images with uniform textures models [13]. However, those model-based methods depend on the determination of prior information, and also need a lot of training. Hence, the generalization ability of those methods are limited.
Different from model-based methods, feature-based methods mainly concentrate on distinguishing and removing the shadow by contour, brightness, color, texture and other features of pixel which are less affected by environmental factors. Hence, those methods have a wide range of application. For example, Xu obtained the stable shadow elimination results through HSV color features, by using the difference idea of image Log domain [14]. Park used shadow depth map and illumination invariance feature to remove shadows [15].
Li proposed a shadow weakening algorithm based on brightness and texture features without the prior training and manual intervention [16]. Salvador proposed a new cast shadow segmentation algorithm based on the shadow spectrum and geometric characteristics of shadows in the scene [17]. All the performance of shadow removal improved by using one or more features of images, but the computational cost is too expensive to satisfy the real time surveillance for motion object. Tang proposed a low computational cost algorithm to remove shadow according to the differences in foreground and background of the composition of pixel gray feature [18]. Chen further proposed a state-of-the-art shadow removal method: RGB pixel ratio comparison (RGB-PRC) method, based on the similar pixel change features. In this method, the shadow can be distinguished and removed directly according to the ratios of pixels between Red, Green and Blue (RGB) channels in the foreground and background [10]. Therefore, the effect of shadow removal can be greatly improved.
In this paper, we also concentrate on distinguishing and removing the shadow by pixel features. Different from the aforementioned features, the proposed pixel feature is obtained according to both the principle of shadow formation and the statistics of a large number of actual scenes. Hence, the feature proposed in this paper is more typical and has wider applicability. All of the above will be discussed in the next sections.

RGB pixel differential correction feature
In this section, RGBP-DC feature is discussed, in comparison with the RGB pixel ratio (RGBP-R) feature proposed in [10]. Generally speaking, given a real pointp, let LðpÞ and L s ðpÞ represent the illuminance reflected from this point with and without the direct light exposures, respectively. In other words, L s ðpÞ represents the illuminance reflected fromp when it is in shadow. Assuming that the coordinate of the corresponding pointp in imaging plane is (x, y), the pixel values of this point in R, G and B channels are denoted as R(x, y), G(x, y) and B(x, y), respectively, and the ones in shadow are denoted as R s (x, y), G s (x, y) and B s (x, y), respectively. According to [10], the aforementioned RGB pixel ratios Ratio R (�), Ratio G (�) and Ratio B (�) are defined as: where RGB s (x, y) ≜ {R s (x, y), G s (x, y), B s (x, y)}. Hence, based on this feature that the RGB pixel ratios with and without the direct light exposure are equal, the shadow area can be distinguish from the object and removed as other background. This completes the analysis of Observation 1. However, for most image sensors, the aforementioned linear relationship for imaging is only valid in a certain light intensity range [19,20]. If light intensity is out of this range, for example the light is severely occluded in strong sunlight, the linear relationship for imaging cannot be guaranteed.  , y)). In other words, in the R channel, the pixel ratios with and without the direct light exposure are commonly unequal. These issues are also be found in the G and B channels. Hence, in the nonlinear range, the RGBP-R feature no longer exists.
This completes the analysis of Observation 2.
In order to eliminate shadows more effectively and robustly, this paper mines a new image feature, i.e., the RGBP-DC feature, to adapt to most shadow removal situations. In Eq (7), Hence, Eq (7) Because s r (C, (x, y)), s g (C, (x, y)), s b (C, (x, y) Therefore, under the linear correction with Eq (16), the differences of ΔR(x, y), ΔG(x, y) and ΔB(x, y) are very small. That is: DDR&Gðx; yÞ ¼ DRðx; yÞ À ½M G2R þ N G2R DGðx; yÞ� ! 0: Given any small threshold T, it is easily to find out that: DDG&Bðx; yÞ < T; ð19bÞ This is the RGBP-DC feature, which can be used to discover and remove the shadow areas.

Differential correction based shadow removal method
In this section, the proposed DC-SR method is described in detail. Firstly, based on ISTD dataset [3,4], a set of parameters in Eq (18) is determined for surveillance environments under common daylight. Secondly, the structure of shadow removal method is designed and the algorithmic complexity is discussed.

Parameters estimation according to ISTD dataset
As seen in Eq (16), constants N B2R , M B2R , N B2G , M B2G , N G2R and M G2R can be learnt as the unknown parameters, given known ΔR(x, y), ΔG(x, y) and ΔB(x, y) under actual monitoring scenes.
A major surveillance scene is monitoring during the day or under sunlight lamps. The light source for this monitoring is sunlight. This paper uses the ISTD dataset, in which all of images are taken under sunlight, to estimate those unknown parameters in such scene. Specifically, as seen in Fig 1, there are three kinds of image in each triplet of ISTD dataset: shadow image, shadow mask image and shadow-free image. To obtain a stable relationships between ΔR(x, y), ΔG(x, y) and ΔB(x, y), this paper derives the values of N B2R , M B2R , N B2G , M B2G , N G2R and M G2R based on the statistics of all triplets in this database.
First, for the ith triplet, the D � RðiÞ, D � GðiÞ and D � BðiÞ, i.e., the means of all differences between the pixel values without and with shadow in shadow area of each R, G and B channel are calculated as: where P s is the pixel set of images of the ith triplet, p is the pixel in P s , N is the number of pixels in P s . R sf (i, p), , p) and B s (i, p) are the values of p in R, G and B channels of shadow-free image and shadow image, respectively. M(i, p) is the value of p in shadow mask image.
To simplify the expression of means of differences, this paper uses D � R, D � G and D � B as common notations for the means of differences of any triplet. As seen in Eq (16) Hence, according Eq (21), the RGBP-DC feature for common daylight monitoring is derived.

The structure of DC-SR method
Generally speaking, the goal of shadow removal for monitoring is to eliminate the effect of shadows on object recognition. When objects are detected by cameras, the shadows are also detected as part of the objects, thus seriously affecting the accuracy of detection. Because there is no RGBP-DC feature in the actual object areas of foreground, the shadows areas can be found and distinguished from object areas by RGBP-DC feature.

PLOS ONE
Specifically, based on the RGBP-DC feature, the structure of the proposed DC-SR method is described in Fig 3. As we can see in this figure, a background image I b need to be firstly determined before monitoring. Then, given a foreground image I f , the absolute value of the first difference of the complete image ΔI c is calculated as follows, Obviously, jΔI c j includes R, G and B channels, i.e., jΔR c j, jΔG c j and jΔB c j. In addition, the grayscales of foreground and background are calculated, and the mask of objects with shadow in foreground image, which is denoted as I m , is derived by thresholding the absolute differences of grayscales. Then, the first order differences of objects with shadow in R, G and B channels are calculated as follows, j DB j¼j DB c j I m : ð23cÞ https://doi.org/10.1371/journal.pone.0276284.g002

PLOS ONE
words, the time complexity of this method is linearly related to the number of pixels of the video frame.

Experiments
In this section, our DC-SR method is evaluated in both outdoor and indoor (with sunlight lamp) environments, in comparison with RGB-PRC method and the Gray Levels Comparison (GLC) method [18]. Further, we test our DC-SR method under real-time monitoring, in order to prove its reliability and stability. For a fair comparison, the thresholds used in all the aforementioned methods are fixed. Specifically, the threshold used in our DC-SR method is set to be 8. According to [10,18], the thresholds used in RGB-PRC and GLC method are set to be 0.008 and 35, respectively.

The implementation description
In this paper, we use Hikvision 2 megapixel USB camera to take photos and use ordinary computer to run the program of the proposed method.

Analysis of shadow removal performance in static scene
Outdoor environment. We compare the shadow removal performances of the aforementioned three methods in outdoor environment. The results are shown in Fig 4. In this figure, the first column shows the background image, the second column shows the foreground image, the third column shows the foreground image with object circled in red lines, the forth to sixth columns show the shadow removal results of the GLC, RGB-PRC and DC-SR methods, respectively. In this bright outdoor environment, the backlit sides of the objects are very dark, which are very similar as the shadows in terms of the intensity of light reflection. Hence, those methods, such as the GLC method, which distinguish shadows relying on the intensity of light reflections will fail. This problem can be obviously seen in the forth column of Fig 4. Moreover, as mentioned in RGB Pixel Differential Correction Feature Section, the RGBP-R feature is not accurate enough to distinguish the shadow area from the object area in those bright light environments. Hence, as seen in the fifth column of Fig 4, the shadow removal performance of RGB-PRC method degrades in those environments. That is, if the object can be completely detected, the shadow cannot be perfectly removed, for example the image in the third row and fifth column. Conversely, if the shadow can be perfectly removed, the object cannot be completely detected, for example the image in the sixth row and fifth column. However, the shadow removal results of the sixth column of Fig 4 show that the proposed DC-SR method can accurately detected the object while well removing the shadow. Hence, our DC-SR method can outperforms other shadow removal methods in outdoor environment with bright light.
Indoor environment. As seen in Fig 5, both the performance of GLC and RGB-PRC are improved because the light intensity is much weaker than sunlight. But obviously, the proposed DC-SR method still provides the most accurate object detection results with similar shadow removal performances.
Evaluation metric. To further verify the shadow removal effect of DC-SR method, we propose the average of object and shadow detection accuracies as the evaluation metric. specifically, the essence of shadow removal is to distinguish the shadows from objects. In other words, the objects need to be correctly detected while well removing the shadows. Hence, we average the object and shadow detection accuracies to obtain a proper overall merit for shadow where N detected object and N detected shadow are the numbers of pixels of detected object and shadow areas, respectively. N actual object and N actual shadow are the numbers of pixels of actual object and shadow areas, respectively. To use the metric (26), actual object and shadow areas need to be known first. Hence, we manually marked the counters of the object for each case, as seen in the third columns of Figs 4 and 5, to get the actual object area. Further, we eliminate the actual object area from the difference image of foreground and background to obtain the actual shadow area. The comparison results are shown in Table 1. The cases of outdoors 1 to 6 corresponds to each row of Fig 4 and the cases of indoors 1 to 3 corresponds to each row of Fig 5. In Table 1, we can see that, in each case, the proposed DC-SR method improves the average of object and shadow detection accuracies by at least 10% except Indoors 3. But in fact, the average accuracy of proposed method is still higher that other methods in the Indoors 3 case. Hence, our DC-SR method outperforms other state-of-the-art shadow removal methods.

Performance testing for monitoring
To further verify the performance of our DC-SR method for monitoring, we test our DC-SR method under real-time monitoring. The results are shown in Figs 6 and 7. In the first rows of the two figures, the blue tracking boxes correctly frame the object area without the shadow area at any time steps. Moreover, in the second rows of the two figures, the white areas are the objects detected after shadow removal. Obviously, we can see that no shadow areas are included and the car was correctly detected. Meanwhile, we test the computational time of all the aforementioned methods with the same Intel Core i7-8700 CPU at 3.2 GHz and 32 GB RAM. The results are that the proposed  DC-SR method consumes 19.9ms for each removal calculation while the GLC and RGB-PRC methods consume 5.0 and 20.9 ms, respectively. Hence, the computational efficiency of the proposed DC-SR method meets the requirements of monitoring, which can be further verified that no frame drops were found in real-time monitoring experiments. Therefore, our DC-SR method can achieve efficient and accurate shadow removal in real-time monitoring.

Conclusion
In this paper, we propose a new differential correction based shadow removal (DC-SR) method based on the new RGB pixel differential correction (RGBP-DC) feature in the shadow areas. From the effect of shadow removal, the proposed RGBP-DC feature, which can well distinguish the shadow areas from objects, is more suitable for shadow removal under both daylight and sunlight lamp environments. Experiments proves that our DC-SR method performs better in comparison with the state-of-the-art shadow removal methods of monitoring. Further, the results of time complexity analysis and algorithm testing in real-time monitoring show that our DC-SR method has the ability to efficiently and accurately remove shadows.
In fact, the performance of our DC-SR method is closely related to the parameters in (16). Although those parameters are set based on the ISTD dataset which covers the main daylight environments and represents the most common relationship between shadow and shadowfree images, in some special low-light or polarized environments, the performance of our method will still degrade. How to improve the adaptability of the method to those special environments will be an important future work.